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INCORPORATING HANDWRITTEN NOTATIONS INTO AN ELECTRONIC DOCUMENT 

APPLICANTS: DHANANJAY KESKAR AND MIC BOWMAN 

BACKGROUND 

The invention relates to incorporating handwritten 
notations into an electronic document. 

Documents such as reports, letters and slide presentations 
are often developed using an iterative review process. Such a 
process can include, for example, sequential or parallel reviews 
of the document by various participants. Comments, suggestions, 
and questions can be generated by each participant and can be 
provided in the form of handwritten notations on a printed 
version of the document. 

Similarly, printed documents can be useful to supplement 
material reviewed during presentations, discussions, meetings or 
other verbal communications. Participants may take notes, write 
questions, or highlight different parts of a handout while 
participating in the associated verbal communication. 

20 BRIEF DESCRIPTION OF THE DRAWINGS 

FIG. 1 is a system block diagram and a document with 
printed text . 

FIG. 2 is a flowchart outlining a users interface with a 
system. 
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FIGS. 3A and 3B are views of a writing/ scanning utensil. 

FIG. 4 is a flowchart outlining functions of a 
writing/ scanning utensil and a computer. 

FIG. 5 is another implementation of a system block diagram 
5 and a document with printed text . 

FIG. 6 is a flowchart outlining functions of an alternative 
implementation of a writing/scanning utensil. 



DETAILED DESCRIPTION 

p 

IOD The system 100 of FIG. 1 includes a writing utensil 102 

capable of storing a writing medium, for example ink or lead, 

H 

^ and dispensing it onto a surface, such as a page 104 of a 

111 

• document. The tip 106 of the writing utensil 102 dispenses the 

9 

writing medium to allow an individual to apply notations to the 

Hi 

Ife page 104. The writing utensil 102 also can capture notation 

13 

data for storage or further processing. Techniques for 
capturing notation data can include determining whether the tip 
106 of the writing utensil 102 is in contact with the page 104 
and tracking the writing utensil's 102 movement on the page 104. 
20 This can be accomplished, for example, by implementing echo- 
location techniques using radio frequency transmission or by 
using image processing techniques in conjunction with a page 
that includes a grid for tracking the motion of the writing 
utensil 102. An integral optical scanner 108 is positioned near 
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the tip 106. The writing utensil 102 also includes a memory 
storage device 110 for storing images captured by the optical 
scanner and captured notation data. 

The page 104 includes text 112 which may be a letter, a 
5 report, or any other printed document. The page 104 also could 
include an illustration, A unique page identifier 114, 
formatted as a bar code, is provided in the lower right hand 
corner of the page 104. Alternatively, the page identifier 114 
could be an alphanumeric . code affixed to the page 104. 
100 A computer 118 including a processor 12 0 and a memory 

m 

storage unit 122 is electrically coupled to an adapter 124 
through a communications channel 126. The adapter 124 is 

O 

IsJ 

configured to mate with the writing utensil 102 in such a manner 

o 

|S as to enable the transfer of data. 

m 

l^n The computer 118 also is electrically coupled to a disk 

13 

drive 128 through a second communications channel 130. The disk 
drive 128 can receive a disk 132 containing computer-executable 
instructions 134 for the computer 118 to perform operations 
described below. The computer 118 can be coupled to other 
20 devices and be configured to read computer-executable 

instructions from computer readable mediums through those 
devices . 

As indicated by FIG. 2, an individual scans 202 the page 
identification data 114 associated with the printed page 104 
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using the writing utensil 102. This may be done by positioning 
the scanner 108 of the writing utensil 102 near the printed bar 
code 114. Alternatively, the individual can scan a document 
identification number and a separate page identification number. 
5 The scanned page identification data 114 is stored in the memory 
storage device 110 of the writing utensil 102. 

An individual uses the writing utensil 102 to apply 204 a 
notation or multiple notations to the printed page 104. The 
notation could be, for example, a note, a question, or a sketch. 

Q 

100 The writing utensil 102 captures the notation data. The scanner 
^0 108 scans the area of the page near the notation as it is being 

.2 applied. The notation data and an image of the surrounding area 

111 

are stored in the memory storage device 110. The writing 

O 

i« utensil 102 can create an association between the notation data 

l^Q and the scanned image. 

Q 

frf, The individual transfers 206 the stored page identification 

data 114, the stored image and the stored notation data to the 
computer 118 by coupling the writing utensil 102 with the 
adapter 124. The writing utensil 102 can include an electrical 
20 contact positioned to mate with a corresponding electrical 
contact on the adapter 124. The data passes through the 
connection. The data then passes from the adapter 124 to the 
computer 118 through the communications channel 126. The 
computer 118 stores this received data in the memory storage 
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device 122. The processor 120 of the computer 118 can 
incorporate the received data into an electronic notation and 
indicate an association between the electronic notation and a 
corresponding passage of an electronic version of the text 112 
in the document . 

After transferring the data, the individual obtains 2 08 an 
electronic version of the printed page that includes 
electronically formatted notations corresponding to the 
handwritten notations applied to the printed version of the page 
104. The electronically formatted notations can be associated 
with specific terms, phrases or passages of the page 104. Such 
an association may be indicated, for example, by a comment 
inserted into* a word processing program, where the comment is in 
a digital format and is linked to a specific passage. 
Alternatively, a list of comments including corresponding 
document numbers, page numbers and/or line numbers referencing 
various sections of a document, can indicate such an 
association. The updated electronic version of the page 104 
including notations can be printed, saved or further modified. 

As shown in FIGS. 3A and 3B, the writing tip 106 of the 
writing utensil 102 extends from approximately the center of a 
front surface of the optical scanner 108 at one end of the 
writing utensil 102. The writing tip 106 is located at the end 
of a cartridge 300 that stores the writing medium. The writing 
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tip 106 dispenses the writing medium from the cartridge 300 when 
the tip 106 contacts a surface of a document. The optical 
scanner 108 is positioned so that it can scan an area of a page 
near a notation being applied by the writing tip 106. Such an 
5 area can include printed data, such as words, letters, marks, 
drawings or bar codes . These scanned areas can be considered 
passage identifiers. 

The writing utensil 102 includes a conductor 302 that 
carries data between the optical scanner 108 and a processor 
lOiQ 304. The conductor can be electrical or optical. The processor 

i 

C 304 is configured to receive data from the conductor 302 and 

J store it m a memory unit 306. The processor 304 can perform 

O 

image-processing functions to create and modify images based on 

i 

!f data received from the scanner 108. The processor 304 also can 



15fg perform processing of captured notation data, for example, to 

p 

1^ create a bitmap image and can establish a correlation between 
specific notation data and a scanned image. Such images may be 
stored in the memory unit 306. The writing utensil 102 also 
includes a wireless transceiver 308 for transmitting and 
20 receiving data communicated between writing utensil 102 and a 
remote device such as the computer 118. A battery 310 is 
provided to support the operations of components in the writing 
utensil 102. A docking station 312 is provided to mate with the 
adapter 124 and includes a contact that can be mated with a 
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complementary battery charging connection on the adapter 124- 
The docking station 312 also includes a second contact that can 
mate with a complimentary data contact of the adapter 124 for 
transferring data to the adapter 124 or other external device. 
5 Referring to FIG. 4, the optical scanner 108 scans 400 page 

identification data 114 printed on the page 104. This page 
identification data 114 travels through the conductor 3 02 to the 
processor 3 04 which stores 4 02 the data in the memory storage 
unit 306. 

O 

lO-p^ When the writing tip 106 dispenses 404 writing medium onto 

\Q the page 104, the writing utensil 102 captures 406 stroke data 

"'4 associated with the notation being applied and the optical 
□ 

W scanner 108 scans 408 an area of the page near the notation 
representing an associated passage. The processor 304 stores 

15'^ 410 the captured stroke data and stores an image of an area near 
jT^ the notation in the memory storage unit 306. Alternatively, the 
optical scanner 108 can be used to scan a portion of a passage 
that is underlined or circled by the writing utensil 102 to 
facilitate identifying an association between that portion of 

20 the passage and a particular notation. The writing utensil 102 
may include a first operational mode used to scan an associated 
passage on a page without applying a notation and a second 
operational used to apply a notation to an area of a page 
without scanning. 

7 
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The writing utensil 102 transfers 414 the stored data to 
the computer 118. The data can be transferred through the 
contacts of the docking station 312 or by employing wireless 
transmission technology using a transceiver device 308. 
5 The computer 118 receives 416 the data from the writing 

utensil 102 and stores the data in the memory unit 122. Based on 
the received page identification data 114, the processor 120 
identifies 418 an electronic version of the printed page stored 
in the memory storage device 122. The identification can be 

O 

lO^g accomplished by correlating the received page identification 

i.rw 

^0 data 114 with a file name associated with the electronic version 

'4 of the page . 
O 

^« The computer 118 creates 420 a correlated electronic 

3 

|p notation associated with a specific passage of the electronic 
15;~ version of the page. The processor 120 deciphers 422 which 
|2 portion of the received image represents the notation and which 
portion of the received image represents printed text or other 
printed matter. The processor 12 0 can convert 424 the portion 
of the received image identified as representing printed text 
20 into digital text. That conversion can be accomplished using 

optical character recognition techniques. The processor 120 can 
then perform 426 a text search of the electronic version of the 
page stored in the memory storage device 122 to identify a 
passage containing the digital text. Alternatively, the 
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processor 120 could employ 428 pattern recognition techniques 
and sub- image pattern recognition techniques to identify a 
passage of the electronic version of the page having patterns 
that substantially match patterns of the received image. 
5 The processor 120 creates 430 an electronic notation based 

on the captured notation data. In some implementations, this 
can include creating a bitmap image or converting the notation 
data into digital text using handwritten character recognition 
techniques . 

□ 

10.?:^ The processor 120 identifies 432 the correlation between 

IS 

the electronic notation and the identified passage of the 

h. a 

'•^1 electronic version of the page, for example, by inserting a 
Q 

ilJ comment into a word processing document. 

0 The computer 118 can save, print or further modify 434 the 

m 

pa 

15^ ~ updated electronic version of the page with the notation data. 

''0 

n 

The writing utensil 102A of Fig. 5 includes a wireless 
transceiver 500 capable of communicating with a second wireless 
transceiver 502 of the computer 118A over a wireless 
communications channel 504. This arrangement can eliminate the 
20 need for a docking station 312 on the writing utensil 102 and 
can eliminate the need for an associated adapter 124 coupled to 
the computer 118A. Data can be transferred immediately to the 
computer 118A as it is scanned by the writing utensil 102A. 
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The writing utensil 102A includes a power switch 506 that 
can disconnect power from the optical scanner 108. The optical 
scanner 108 can be turned off at certain times to allow the user 
to apply notations to a page without having the notations 
5 scanned and saved into memory. Using the power switch 506 in 
this manner can help ensure that the available writing utensil 
memory is used efficiently and can extend the life of the 
writing utensil's battery 310. The power switch 506 also can 
extend the time it takes to completely fill available memory of 
10;^ the writing utensil 102A. 

i 

The writing utensil 102A includes a memory indicator 508 to 
indicate when available memory drops below a predefined level. 

Q 

y A charge indicator 510 is provided to indicate when the charge 
O in the battery 310 drops below a specified level. These 

m 

15^^ indicators 508, 510 can include, for example, light emitting 
!^ diodes (LEDs) . 

As shown in FIG. 6, the writing utensil 102A can scan 602 
page identifying data 114 and transmit 604 it to a remote 
device, such as the computer 118A. The writing utensil 102A can 

20 apply 606 a notation to the page 104 while simultaneously 

capturing 608 and transmitting 610 the stroke data as well as 
scanning 612 and transmitting 614 the passage identification 
data 114. The writing utensil 102A can establish a correlation 
between the captured stroke data and the scanned passage 
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identification data. Upon receiving the transmitted data from 
the writing utensil 102A the computer 118A can operate in a 
manner similar to that described above with reference to the 
computer 118 of FIG. 4. The computer 118A can process the 
5 received data as soon as it is received or can store it for 

subsequent processing. Implementation of wireless transmission 
techniques can reduce the time required to create an updated 
electronic document including notations. 

Systems implementing the techniques described above may 

Q 

IOIq provide one or more of the following advantages. Handwritten 
comments, notes or suggestions on a printed document may be 

^. s 

'-J incorporated automatically and quickly into an electronic 

o 

y version of the text of the document. That can facilitate 

^•5 conducting the extensive review processes that are often 
IP 

15 involved in preparing various documents. Documents being 

n 

1^ prepared for finalization or issue by an organization or 

individual can be updated relatively quickly, and a multilevel 
review process for a document can be expedited. Documents can 
be reviewed at any location and at any time using a pen and 
20 paper approach and comments, suggestions and changes can be 

incorporated rapidly into an electronic version of the text of 
the document . The techniques can enable a user to incorporate 
handwritten comments to a printed document into an electronic 
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version of that document without requiring subsequent typing of 
the comments. 

Modifications of the techniques described above are 
possible. For example, the writing utensil need not include the 
5 integral memory unit 110 or the wireless transceiver 500. The 
computer 118 can track data received from different pens and 
different document reviewers and customize each set of 
electronic notations received in different colors or fonts to 
indicate comments received from different people. The writing 

O 

lOig utensil 102 and the computer 118 may perform the tasks described 

5 

kQ above in a different order. The electronically formatted 
''"4 comments can be inserted into word processing documents, 

o 

W spreadsheet documents, slide presentation documents, computer 

5 

:^ aided drafting documents or other types of electronic documents. 

15',^ The battery can be replaceable. The scanner can be removable to 
1^ allow the writing utensil to be used occasionally for writing 
purposes only. The writing medium cartridge 3 00 also can be 
replaceable. Removable and replaceable memory storage devices 
110 may be used with the writing utensil so that when one memory 

20 storage device 110 becomes full, a user can remove it and 

replace it with a new one. The writing utensil also can allow a 
user to erase data stored in the integral memory storage device 
110. 
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Various features of the system can be implemented in 
hardware, software or a combination of hardware and software. 
For example, some features of the system can be implemented in 
computer programs executing on programmable computers. Each 
5 program can be implemented in a high level procedural or object- 
oriented programming language to communicate with a computer 
system. Furthermore, each such computer program can be stored 
on a storage medium, such as read-only-memory (ROM) readable by 
a general or special purpose programmable computer or processor, 




medium is read by the computer to perform the functions 



described above . 



for configuring and operating the computer when the storage 



Other implementations are within the scope of the following 




claims . 
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